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Abstract 


The new curriculum of medical schools has brought the need to develop a professional committed to ethics, 
capable of understanding the role of social factors, the insertion of the individual into the family and their 
role in promoting health. These skills and the use of new methodologies in teaching-learning process have 
demonstrated that the traditional performance assessment methods are limited in holistically evaluating 
the student. This study analyzed the correlation among different methods for evaluating competencies in 
medical internship students and Objective Structured Clinical Examination (OSCE). A cross-sectional, 
descriptive and analytical study was performed among 5" year medical students at Pediatric, Surgery, 
Medical Clinic, Family and Community Medicine and Gynecology and Obstetrics. Grades of each student 
obtained through different evaluations (theoretical test, conceptual evaluation and test skills) at the end 
of each rotation were correlated with the individual results from OSCE. In regard to correlations with the 
OSCE score values, it was found a weak to moderate positive and statistically significant relationship with 
the global cognitive score (r= .22, p< .001), global abilities score (r= .26, p< .001) and with the global 
behavioral score (r= .38, p< .001). The students' performance in the various assessments of performance 
evaluated here was positively correlated to their performance on OSCE. Furthermore, the organization 
of an OSCE-type assessment prepared in multiple, committed and competent hands may serve as a unique 
assessment and, in fact, assess the development of clinical skills for future doctors. 
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Introduction 


The teaching and learning process in higher education has progressed remarkably in 
recent years worldwide, including in medical schools (Irby et al., 2010; Mann, 2011). In direct 
alignment with this new pedagogical reality, Brazilian medical schools have been re-organizing 
their curricula by competencies and adapting to the implementation of new teaching strategies 
with different learning scenarios, incentives for research training and reassessment of evaluation 
methods (Santos, 2011). 

In order to achieve all the points described in the National Guidelines (Conselho Nacional 
de Educagao, 2014), the curriculum of medical courses should focus on making students able to 
develop more than theoretical skills during their training. Thus, professors must instruct future 
professionals committed to ethics, able to understand the role of social factors in maintaining 
health, the insertion of the individual in the family, their role in promoting health and adequate 
communication with the patients and their families. Therefore, adjustments to the knowledge 
pyramid suggested by Miller (1990) have been proposed, where, at the top of the pyramid, the 
student "consistently demonstrates the attitudes, values and behaviors expected of those who 
came to think, act and feel like a physician" (Cruess et al., 2016; Miller, 1990). Pedagogically, 
for this new professional model, changes in teaching strategies have become imperative. With 
more innovative curriculum design based in competencies, the adoption of active learning 
strategies have been recommended (Frohna et al., 2006; Prado et al., 2011; Schmidt et al., 
2009). 


Research Focus 


Another crucial point is: how to evaluate students’ performance in this new scenario? 
It is still widely accepted that assessment of academic performance is necessary to monitor 
student’s development. Such assessment serves as a guide for assessing individual progress in 
incorporating the knowledge and skills necessary for medical profession. However, choosing 
an appropriate and efficient assessment method is not an easy task and remains debated 
(Lurie et al., 2009). Traditional approaches using cognitive assessment provide an inaccurate, 
incomplete picture of the holistic training, not being in alignment with the guidelines of the new 
methodologies proposed in competency-based curricula (Troncon, 2004). A more complete 
assessment should do more than measure the volume of correct information, it should be able to 
provide the desired evidence that students are achieving curricular objectives and competencies. 

Of note, there are different types of assessment approaches and professors should always 
take into consideration which one is the most appropriate to assess whether the objectives of 
a given course and/or discipline have been achieved by students. The Structured Objective 
Clinical Examination (OSCE) is one of these assessment tools already successfully tested in 
different courses, including medical schools (Carraccio & Englander, 2000; Fischer et al., 2017; 
Kolivand et al., 2020). It is designed to test, in a practical way and in a controlled environment, 
the performance of clinical and main medical skills. Although it has already been tested and 
applied at major universities in the world such as in Canada, Germany, United Kingdom and 
the United States (Chisnall et al., 2015; Furmedge et al., 2016; Goldhamer et al., 2018; Lafleur 
et al., 2015; Miiller et al., 2019; Stansfield et al., 2016), it is still not widely applied in Brazilian 
Medical schools in detriment of traditional assessments (Troncon, 2004). 

In this context, it is interesting to note that recently in Brazil, some government programs 
in the areas of health and education have provided an explosive growth in the number of medical 
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courses. In these new medical schools, in the face of anew society driven by technology, teaching 
strategies have been rethought and adequate (Farias et al., 2015). Although, these schools 
proposed mostly innovative curricula based on interdisciplinarity and multi-professionalism 
with the use of problem-based learning (PBL) and Student-centered learning (Meireles et al., 
2019; Takenami et al., 2018), the OSCE-type evaluation methods are still not widely used. 


Literature Review 


The Objective Structured Clinical Exam was initially proposed in the 1970s to measure 
the achievement of clinical skills (Harden et al., 1975). Since then, it has been applied (with 
variations) in an increasing number of medical schools worldwide (Carraccio & Englander, 
2000; Patricio et al., 2013). In this scenario, this type of assessment and its variations have 
already been tested in different health courses, from basic sciences such as physiology and 
anatomy to more advanced units in the clinical axis such as urology, gynecology and even 
radiology (Dumont et al., 2016; Noureldin et al., 2015; Van Den Berk et al., 2011; Wani, 2015; 
Yaqinuddin et al., 2013). 

The organization of the OSCE may vary between the institutions that apply it as an 
evaluation method, however the principles that underlie it are common among schools. In 
summary, students perform sequentially a series of cases and clinical stations structured to 
measure the achievement of skills such as (1) communication and professionalism, (11) obtaining 
a history, (111) physical examination, (iv) clinical reasoning and (v) practical / technical skills. 
Different strategies can be used in the stations, for example, the interaction with a simulated 
patient, a computer simulation, a series of clinical information etc. Each of these stations has a 
professor to assess student performance based on a pre-defined checklist. Studies show that if 
there is a compromise in the elaboration of the OSCE, it becomes a workable, valid, reliable, 
cheap and efficient method (Gormley, 2011; Zayyan, 2011). 

It is important to carry out a critical analysis of this evaluation strategy. There is still 
a great deal of heterogeneity between models from different medical schools and even among 
the evaluators that participate of the evaluation process (Guraya et al., 2010; Schleicher et al., 
2017). In addition, the experimental designs of the studies that evaluate the efficiency of the 
OSCE, in general, have important methodological limitations which make us take a cautious 
look at the subject which certainly needs a better exploration (Turner & Dankoski, 2008). 

A point that must be highlighted, however, is the positive evaluation on the part of 
the students who consider the method efficient and perceive its advantages, despite finding it 
very stressful (Khorashad et al., 2014). Another important point is that the OSCE has already 
been tested in comparison with other assessment models demonstrating inconclusive levels 
of efficiency and perceptions (Chisnall et al., 2015; Lakshmipathy, 2015; Turner & Dankoski, 
2008). 


Research Aim 


In this context, in order to contribute to the OSCE's validation as an effective evaluation 
model, it is essential to compare it with the other assessment models. In the present research, in 
students from a Brazilian private medical school, it was assessed the correlation of evaluation 
scores originated from different assessment methods commonly used in medical schools with 
scores from OSCE. 
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General Background 


A Cross-sectional, descriptive and analytical study was conducted with medical students 
in order to assess the correlation between their performances in OSCE and other types of 
evaluation tests. For this purpose, their scores in five different curricular areas of internship 
were analyzed. 


Sample 


One hundred ninety-eight students of the 5th year of Medicine course (Internship), 
in a private medical school from Salvador, Northeast Brazil. The analyzed curricular areas 
were “Pediatrics”, “Surgery”, “Internal Medicine’, “Family and Community Medicine” and 
“Obstetrics and Gynecology”. In this medical school, each student completes a two-month 
internship in each of the aforementioned areas developing activities at a training hospital, 
outpatient clinic, emergency unit, surgical and obstetric center, and Family Health Units. The 
variables used for analyses were epidemiologic data and grades of each student in distinct 
evaluations described below which were performed at the end of each rotation. The grades 
could range from 0 to 10 and have 1 decimal place. 


Procedures and Data Collection 


In order to test the cognitive domain, a theoretical and objective test was applied with 
20 multiple-choice questions, dealing with the most common subjects in each area of medical 
knowledge. The main professor of each curricular unit prepared the test. 

To test the ability skills, it was applied the Objective Structured Practical Exam (OSPE) 
model on low complexity simulators for the curricular units of Surgery and Pediatrics, preserving 
patient safety. Invasive, noninvasive procedures and clinical situations, analysis of laboratory 
and imaging tests were tested. The test was applied in the Simulation Center and graded each 
student according to the standardized form prepared by professors of the respective areas. For 
the curricular units of Internal Medicine, Family and Community Medicine, and Obstetrics and 
Gynecology, we used the mini-clinical evaluation exercise (Mini-Cex) (Megale et al., 2009). 

With the purpose of test, the behavioral skills, the preceptor of each internship field 
performed a conceptual evaluation, following a standardized form, representing the student’s 
performance during the 2-month rotation. The final grade to each student was the weighted 
average of 30% performance in cognitive assessment, 30% performance in abilities skills and 
40% performance in the conceptual evaluation. 

At the end of the internship, an OSCE was carried out among all the students. It was 
composed by stations of clinical attendance with actors, representing habitual clinical situations 
in Clinical Medicine, Family and Community Medicine and Surgical Clinic, a station of 
case-long Pediatrics and a video station on ethical conduct of diagnosis and information to 
the relatives of HIV patients. Professors specialized in each area of knowledge, graded each 
student, following a standardized single file. 

All students, enrolled in the 5th year of the Medical course, performed all the evaluations 
and signed the Free and Informed Consent Form. 
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Statistical Analysis 


The normality of the sample was evaluated by the Kolmogorov-Smimov test 
(“Kolmogorov—Smirnov Test,” 2008). The mean values with standard deviation or median with 
interquartile range were used as measures of central tendency and dispersion, depending if a 
given variable presented or not Gaussian distribution in the study population. The comparison 
between the average grades of the evaluations was done through the Kruskal Wallis test with 
multiple Dunn comparisons or nonparametric linear trend post-test (Kruskal & Wallis, 1952). 

To test the correlation between the assessments, it was used the Spearman correlation 
coefficient (Spearman, 2010). The p-values were considered statistically significant when p < 
.05. The OSCE assessment grade was considered the gold standard. This metric was used to 
study its correlation with the grades of the theoretical test (assessing cognitive competence), 
the marks of the practical test (assessing the abilities skills) and the grades of the conceptual 
evaluation (assessing the behavioral competence). 


Ethical Considerations 


The investigations were conducted according to the principles expressed in the 
Declaration of Helsinki (World Medical Association, 2013). This work was executed in 
accordance with Resolution no. 466/12 of the Brazilian National Health Council and was 
approved by the Institutional Review Board of Salvador University (UNIFACS, Laureate 
International Universities). The students who accepted to participate in the research signed 
Terms of Free and Clarified Compromise. 


Research Results 
Descriptive Results 


Results of 198 students were analyzed. The group comprised 144 female students (73%), 
with a mean age of 25.8+3 years old, of high economic standard. Eighty percent (80%) of the 
students attended their first higher education course, but 20% had started or completed some 
other course previously, mostly in health. 

The average final grade of the students in each of the 5 curricular units (global cognitive, 
global skills and overall behavioral) and OSCE were obtained: the Global cognitive mean of 
7.0£0.7 and the median 7.0; the overall global skills were 8.640.6 and the median 8.6; the 
overall behavioral mean was 9.3+0.3 and the median 9.4; the mean of the weighted average 
grade (WAG) was 8.3+0.4 and the median 8.4; the mean of OSCE was 7.6+0.8 and the median 
7.7. The scatter plots of the grades are shown in Figure 1. 
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Global grades of students. In (A) the mean of cognitive, abilities and behavioral grades are 


presented. In (B) the OSCE and WAG grades are shown 
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*WAG (weighted average grade) = (cognitive 0,3) + (ability x 0,3) + (behavioral x 0,4) 
Analytical Results 


To attest to the student's cognitive domain, it was used the traditional multiple-choice 
test. A weak positive but statistically significant correlation with the OSCE results in the overall 
score was detected (7= .22 p< .001). When analyzing the evaluations performed in each rotation, 
it was observed a correlation in Internal Medicine, Obstetrics and Gynecology and Pediatrics 
(Figure 2A). To test the skill domain, a practical test was applied. It was found a weak and 
statistically significant correlation with the OSCE results in the overall score with r= .26 and 
p< .001. When analyzing the evaluations performed in each rotation, a positive correlation 
in Internal Medicine and Pediatrics was found (Figure 2B). To attest the behavioral domain 
of the student, a conceptual evaluation was performed. It was found a weak and statistically 
significant positive correlation with the OSCE results in the overall score with 7= .38 and 
p< .001. When analyzing the evaluations performed at each rotation, correlation in Internal 
Medicine, Obstetrics and Gynecology and Surgery were found (Figure 2C). Considering the 
final weighted score, we found a strong positive correlation with the OSCE 7= .38 and p< .001 
(Figure 2D). 
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Figure 2 

Distribution per curricular unit of Spearman correlation vs OSCE. The Spearman correlation 
of (A) Cognitive, (B) Abilities, (C) Behavioral and (D) Weighted final grade evaluations with 
OSCE are presented 
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Discussion 


In this research, the average age of the students was 25.843 years old, which puts them 
in the sociological category of Generation Y. In addition, students mostly belong to a higher 
social class. Such characteristics make them widely familiar with technological resources and 
Internet, however, this generation of students is known for their impatience, immediacy and 
difficulty in following conventional models of work and hierarchy (Desy et al., 2017; Nalliah 
& Allareddy, 2014). Pedagogically, these characteristics must be considered in the teaching- 
learning and evaluation process. 

It has been consistently demonstrated that this generation of students adapts well to the 
use of active learning methodologies (Quintanilha et al., 2018). In general, the use of more 
dynamic assessment tools results in better performance compared to more traditional tools. 
In the present study, it was found a lower performance in traditional cognitive assessments 
than in skill (practical or Mini-Cex) and conceptual assessments (Figure 1). This corroborates 
the findings of Sandoval, who evaluated the students' performance in the evaluations of the 
Pediatric Internship. The researcher detected approximately 20% lower scores on written exams 
than the result of observing daily clinical practice (Sandoval et al., 2010). A curriculum based 
on active methodologies may play an important role in these findings, as it stimulates students' 
practical and attitudinal character. 

Here, we can clearly see a narrowing in the distribution of grades for conceptual 
assessment. Despite the use of a standardized instrument and the incentive to individualize the 
concept of each student, the values show low dispersion, with low sensitivity and the ability to 
detect low student performance. It demonstrates that these kinds of evaluations sometimes do 
not correspond to the students' real performance and it is important to think deeply about how 
to assess these cases better. 

An important observation to be highlighted is that most health science professors working 
in the medical courses were not prepared for the teaching activities, reproducing, in general, the 
traditional model in which they were trained. Although institutions may offer update courses 
and pedagogical forums, most medical professors are unable to attend them (Quintanilha et al., 
2020). This reality is observed in other studies (Costa, 2010) contributing to the maintenance 
of the wrong perception that the evaluation is only for approval (Diedrichs et al., 2012). They 
do not consider the main aim to correct the course according to the educational process and 
to act as an important pillar of the teaching-learning process (Furmedge et al., 2016). In this 
context, the continuous development of the professor can promote a paradigm shift, and this is 
extremely important for the construction of a reflective, positive and transformative evaluation 
philosophy. 

Moreover, there is a gap between the recommendation of conducting classes using 
active learning methodologies and the common application of exams using only cognitive tests. 
The implementation of new assessment tools finds it difficult to convince professors about 
the validity of their use since, as previously discussed, they were trained and evaluated in 
traditional ways during their academic life. They complain about the number of checklists and 
bureaucratic formats that competency-based assessment models establish (Lombarts, 2015). 
However, based on continuous practice and the consequent perception of the extension of this 
type of evaluation, there is a decrease in resistance and such changes in evaluation methods can 
prove to be of great value in the process of medical training based on competences and in the 
training of new future physicians. 

In this sense, studies show that the OSCE presents itself as a complete and reliable form of 
assessment in medical schools (Matsuo et al., 2013; Townsend et al., 2001; Walters et al., 2005; 
Zayyan, 2011). In the present research, it was demonstrated that the student's performance in the 
various assessments can be proportional to their performances in the OSCE, with a statistically 
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significant correlation. Our results corroborate the data from the study of Couto (2019) who 
demonstrated that assessment strategies in problem-based learning (PBL) approach, including 
Formative Assessment, Progress Testing and OSCE have positive correlations, which increases 
as the medical course becomes more complex (Couto et al., 2019). 

In this regard, the OSCE, despite the greater complexity of its execution, especially 
in large groups, must be evaluated by medical education institutions as it is a comprehensive 
assessment that effectively measures various degrees of medical skills. This assessment 
modality can even replace a huge range of assessments from different disciplines which can 
have their main contents and competencies covered at once in a single exam. 


Conclusions and Implications 


The definition of efficient forms of assessment is an issue widely discussed in the field of 
education. In Medical education, with the curricular changes that have occurred in the last few 
years, such as the introduction of models such as Problem Based Learning and Active Learning 
strategies, it is incoherent to maintain the same traditional forms of assessment. The Objective 
Structured Clinical Exam (OSCE) has been presented as a useful and effective alternative in 
several Medical schools. In the present study, we compared the performance of students who 
undertook different assessment strategies with their OSCE performance. It was observed that 
there is a weak or moderately positive relationship with statistical relevance between them. 

In practical terms, considering the OSCE's correlation with other assessment models and 
especially the large number of tests that students usually take and the great variability among 
them, the proposal for the organization of an OSCE-type assessment prepared in multiple, 
committed and competent hands, can serve as a unique assessment and, in fact, assess the 
development - or not - of clinical skills for future physicians. 

Since we are evaluating a single group of students from a specific university, the 
results presented may not be extrapolated to other educational contexts. However, there are 
many studies that corroborate with our data making us believe that, in fact, our results are 
representative and suggest a reality in medical courses in general. 

So, there is an urgent need for new, robust and analytical studies that assess the 
effectiveness and perception of learning, as well as application models that increase quality and 
decrease heterogeneity and bias. In this way, medical education educators and managers will be 
able to base their practices on what we call evidence-based medical education. 


Limitations and Suggestions 


As this is an observational study, the results obtained are influenced by the subjectivity of 
the evaluation process, from the perspective of each professor and do not allow us to conclude 
casual relationships. An interesting suggestion is training professors in the standardization of the 
checklists and cognitive tests, in addition to training of each professor that teaches in practical 
setting the importance of the objectivity on behavioral assessments, to analyze whether the 
evaluations make it even closer to the “gold-standard” assessment. 
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